1. Field of the Invention
This invention relates in general to automated testing of software products for computing systems. More specifically, this invention relates to manual reproduction and logging of the operations performed by an automated testing system on a software product.
2. Description of the Background
Producers of software products generally test the software products prior to release to assure its proper operation on as many different operating systems and computer hardware platforms as possible.
In testing a software product, both manual tests and automated tests are conventionally used to verify the proper operations of the software product. Manual tests are generally performed by a person exercising the various features and functions of the software product in the same manner as a user would be expected to use the product. If a large number of tests are to be performed on the software product, manual testing is generally slow and unreliable. Manual testing is therefore generally undesirable given the market and customer demands for producing adequately tested software products in reasonable amounts of time.
Automated testing is a way of testing a software product through the use of test software, operating on a computing system, which simulates actual user input and verifies the proper operations of the software product. The test software seeks to exercise every capacity of the software product as a user would if the user had unlimited time. The software product is tested using multiple variables and inputs on various computing platforms and systems within which the software product may be used. Automated tests generally run faster and are more reliable than a manual test for testing a software product.
As software products become more complex and contain a greater number of routines and features, the test software routines have become more lengthy and complex, requiring increasing amounts of time to setup and perform.
Typically, to develop an automated test for a particular software product, a software test designer develops detailed knowledge of the software product by studying the product specifications, extracting a list of features of the software product, and generating a number of operational expectations which the product should satisfy under certain operating conditions. The test designer then develops a set of "test cases," in the form of test software routines, which exercise the code blocks in the software product to verify its proper operation. The test designer may create hundreds or even thousands of test cases for a single complex software product. A collection of test cases for a software product is known as a test suite.
It will be appreciated that it may require days or weeks to test fully a software product by exercising each code block and each code path within the product on all possible combinations of computer models and operating systems on which the product was designed to be run. As software products become even more complex, the amount of testing time may become even larger, thereby increasing the production costs of the software product.
Despite the benefits of automated testing of software products, conventional automated testing utilities do not generally provide sufficient information to assist a person, other than the test designer, to determine the source of the error when a software product fails an automated test. A simple error message is generally insufficient to fully appreciate the nature and possible cause of the error without detailed knowledge of the actual test steps performed by automated test software. However, only the test designer would generally know enough about the test performed to understand the nature of the product failure since, as previously mentioned, these tests can take days to perform. Interpreting the actual testing software code to trace the possible source of the software product failure can also take substantial amounts of time, particularly if the test designer is unavailable to assist in the review of the product error.
While conventional automated test platforms provide stack traces which can indicate a variety of environmental variables, pointers, counters, and memory values when an error occurs, such information is often too cryptic and difficult to understand by a person who is not intimately familiar with the test software. When a software product fails an automated test, it would also be difficult for a person unfamiliar with the test software to manually reproduce the exact steps performed by the automated test without first having to study and understand the complex automated test software routines.
Further, when testing a new release or new version of an existing software product which itself was already successfully tested, if the new version contains only slight differences over the existing software product, it would be wasteful to write from scratch a completely new set of test cases for the new version of the product, or to run the full cycle of automated tests on the new version, because this process could take days.
What is needed is an automated testing system which provides simple textual descriptions of the test steps performed by the test software so that a person not intimately familiar with the test software can more easily determine the source of the error in the software product.
What is also needed is an automated testing system which provides a concrete list of test steps which would enable a person to manually reproduce the exact steps performed so that the point of failure of the software product can be manually verified.
What is further needed is a software testing method which would provide a specific list of test steps, performed by an automated test of an existing software product, for manual reproduction of the automated test by a person desiring to quickly test the basic functionality of a new software product release or version only which is slightly different than the existing software product.